<!DOCTYPE html>

<html lang="en">
  <head>
    <meta charset="utf-8" />
    <meta name="viewport" content="width=device-width, initial-scale=1.0" /><meta name="generator" content="Docutils 0.19: https://docutils.sourceforge.io/" />

    <meta charset="utf-8">
    <meta name="viewport" content="width=device-width, initial-scale=1, shrink-to-fit=no">
    <meta http-equiv="x-ua-compatible" content="ie=edge">
    
    <title>7.1. Word2vec &#8212; FunRec 推荐系统 0.0.1 documentation</title>

    <link rel="stylesheet" href="../_static/material-design-lite-1.3.0/material.blue-deep_orange.min.css" type="text/css" />
    <link rel="stylesheet" href="../_static/sphinx_materialdesign_theme.css" type="text/css" />
    <link rel="stylesheet" href="../_static/fontawesome/all.css" type="text/css" />
    <link rel="stylesheet" href="../_static/fonts.css" type="text/css" />
    <link rel="stylesheet" type="text/css" href="../_static/pygments.css" />
    <link rel="stylesheet" type="text/css" href="../_static/basic.css" />
    <link rel="stylesheet" type="text/css" href="../_static/d2l.css" />
    <script data-url_root="../" id="documentation_options" src="../_static/documentation_options.js"></script>
    <script src="../_static/jquery.js"></script>
    <script src="../_static/underscore.js"></script>
    <script src="../_static/_sphinx_javascript_frameworks_compat.js"></script>
    <script src="../_static/doctools.js"></script>
    <script src="../_static/sphinx_highlight.js"></script>
    <script src="../_static/d2l.js"></script>
    <script async="async" src="https://cdn.jsdelivr.net/npm/mathjax@3/es5/tex-mml-chtml.js"></script>
    <link rel="index" title="Index" href="../genindex.html" />
    <link rel="search" title="Search" href="../search.html" />
    <link rel="next" title="参考文献" href="../chapter_references/references.html" />
    <link rel="prev" title="7. Appendix" href="index.html" /> 
  </head>
<body>
    <div class="mdl-layout mdl-js-layout mdl-layout--fixed-header mdl-layout--fixed-drawer"><header class="mdl-layout__header mdl-layout__header--waterfall ">
    <div class="mdl-layout__header-row">
        
        <nav class="mdl-navigation breadcrumb">
            <a class="mdl-navigation__link" href="index.html"><span class="section-number">7. </span>Appendix</a><i class="material-icons">navigate_next</i>
            <a class="mdl-navigation__link is-active"><span class="section-number">7.1. </span>Word2vec</a>
        </nav>
        <div class="mdl-layout-spacer"></div>
        <nav class="mdl-navigation">
        
<form class="form-inline pull-sm-right" action="../search.html" method="get">
      <div class="mdl-textfield mdl-js-textfield mdl-textfield--expandable mdl-textfield--floating-label mdl-textfield--align-right">
        <label id="quick-search-icon" class="mdl-button mdl-js-button mdl-button--icon"  for="waterfall-exp">
          <i class="material-icons">search</i>
        </label>
        <div class="mdl-textfield__expandable-holder">
          <input class="mdl-textfield__input" type="text" name="q"  id="waterfall-exp" placeholder="Search" />
          <input type="hidden" name="check_keywords" value="yes" />
          <input type="hidden" name="area" value="default" />
        </div>
      </div>
      <div class="mdl-tooltip" data-mdl-for="quick-search-icon">
      Quick search
      </div>
</form>
        
<a id="button-show-source"
    class="mdl-button mdl-js-button mdl-button--icon"
    href="../_sources/chapter_appendix/word2vec.rst.txt" rel="nofollow">
  <i class="material-icons">code</i>
</a>
<div class="mdl-tooltip" data-mdl-for="button-show-source">
Show Source
</div>
        </nav>
    </div>
    <div class="mdl-layout__header-row header-links">
      <div class="mdl-layout-spacer"></div>
      <nav class="mdl-navigation">
          
              <a  class="mdl-navigation__link" href="https://funrec-notebooks.s3.eu-west-3.amazonaws.com/fun-rec.zip">
                  <i class="fas fa-download"></i>
                  Jupyter 记事本
              </a>
          
              <a  class="mdl-navigation__link" href="https://github.com/datawhalechina/fun-rec">
                  <i class="fab fa-github"></i>
                  GitHub
              </a>
      </nav>
    </div>
</header><header class="mdl-layout__drawer">
    
          <!-- Title -->
      <span class="mdl-layout-title">
          <a class="title" href="../index.html">
              <span class="title-text">
                  FunRec 推荐系统
              </span>
          </a>
      </span>
    
    
      <div class="globaltoc">
        <span class="mdl-layout-title toc">Table Of Contents</span>
        
        
            
            <nav class="mdl-navigation">
                <ul>
<li class="toctree-l1"><a class="reference internal" href="../chapter_preface/index.html">前言</a></li>
<li class="toctree-l1"><a class="reference internal" href="../chapter_installation/index.html">安装</a></li>
<li class="toctree-l1"><a class="reference internal" href="../chapter_notation/index.html">符号</a></li>
</ul>
<ul class="current">
<li class="toctree-l1"><a class="reference internal" href="../chapter_0_introduction/index.html">1. 推荐系统概述</a><ul>
<li class="toctree-l2"><a class="reference internal" href="../chapter_0_introduction/1.intro.html">1.1. 推荐系统是什么？</a></li>
<li class="toctree-l2"><a class="reference internal" href="../chapter_0_introduction/2.outline.html">1.2. 本书概览</a></li>
</ul>
</li>
<li class="toctree-l1"><a class="reference internal" href="../chapter_1_retrieval/index.html">2. 召回模型</a><ul>
<li class="toctree-l2"><a class="reference internal" href="../chapter_1_retrieval/1.cf/index.html">2.1. 协同过滤</a><ul>
<li class="toctree-l3"><a class="reference internal" href="../chapter_1_retrieval/1.cf/1.itemcf.html">2.1.1. 基于物品的协同过滤</a></li>
<li class="toctree-l3"><a class="reference internal" href="../chapter_1_retrieval/1.cf/2.usercf.html">2.1.2. 基于用户的协同过滤</a></li>
<li class="toctree-l3"><a class="reference internal" href="../chapter_1_retrieval/1.cf/3.mf.html">2.1.3. 矩阵分解</a></li>
<li class="toctree-l3"><a class="reference internal" href="../chapter_1_retrieval/1.cf/4.summary.html">2.1.4. 总结</a></li>
</ul>
</li>
<li class="toctree-l2"><a class="reference internal" href="../chapter_1_retrieval/2.embedding/index.html">2.2. 向量召回</a><ul>
<li class="toctree-l3"><a class="reference internal" href="../chapter_1_retrieval/2.embedding/1.i2i.html">2.2.1. I2I召回</a></li>
<li class="toctree-l3"><a class="reference internal" href="../chapter_1_retrieval/2.embedding/2.u2i.html">2.2.2. U2I召回</a></li>
<li class="toctree-l3"><a class="reference internal" href="../chapter_1_retrieval/2.embedding/3.summary.html">2.2.3. 总结</a></li>
</ul>
</li>
<li class="toctree-l2"><a class="reference internal" href="../chapter_1_retrieval/3.sequence/index.html">2.3. 序列召回</a><ul>
<li class="toctree-l3"><a class="reference internal" href="../chapter_1_retrieval/3.sequence/1.user_interests.html">2.3.1. 深化用户兴趣表示</a></li>
<li class="toctree-l3"><a class="reference internal" href="../chapter_1_retrieval/3.sequence/2.generateive_recall.html">2.3.2. 生成式召回方法</a></li>
<li class="toctree-l3"><a class="reference internal" href="../chapter_1_retrieval/3.sequence/3.summary.html">2.3.3. 总结</a></li>
</ul>
</li>
</ul>
</li>
<li class="toctree-l1"><a class="reference internal" href="../chapter_2_ranking/index.html">3. 精排模型</a><ul>
<li class="toctree-l2"><a class="reference internal" href="../chapter_2_ranking/1.wide_and_deep.html">3.1. 记忆与泛化</a></li>
<li class="toctree-l2"><a class="reference internal" href="../chapter_2_ranking/2.feature_crossing/index.html">3.2. 特征交叉</a><ul>
<li class="toctree-l3"><a class="reference internal" href="../chapter_2_ranking/2.feature_crossing/1.second_order.html">3.2.1. 二阶特征交叉</a></li>
<li class="toctree-l3"><a class="reference internal" href="../chapter_2_ranking/2.feature_crossing/2.higher_order.html">3.2.2. 高阶特征交叉</a></li>
<li class="toctree-l3"><a class="reference internal" href="../chapter_2_ranking/2.feature_crossing/3.summary.html">3.2.3. 总结</a></li>
</ul>
</li>
<li class="toctree-l2"><a class="reference internal" href="../chapter_2_ranking/3.sequence.html">3.3. 序列建模</a></li>
<li class="toctree-l2"><a class="reference internal" href="../chapter_2_ranking/4.multi_objective/index.html">3.4. 多目标建模</a><ul>
<li class="toctree-l3"><a class="reference internal" href="../chapter_2_ranking/4.multi_objective/1.arch.html">3.4.1. 基础结构演进</a></li>
<li class="toctree-l3"><a class="reference internal" href="../chapter_2_ranking/4.multi_objective/2.dependency_modeling.html">3.4.2. 任务依赖建模</a></li>
<li class="toctree-l3"><a class="reference internal" href="../chapter_2_ranking/4.multi_objective/3.multi_loss_optim.html">3.4.3. 多目标损失融合</a></li>
<li class="toctree-l3"><a class="reference internal" href="../chapter_2_ranking/4.multi_objective/4.summary.html">3.4.4. 小结</a></li>
</ul>
</li>
<li class="toctree-l2"><a class="reference internal" href="../chapter_2_ranking/5.multi_scenario/index.html">3.5. 多场景建模</a><ul>
<li class="toctree-l3"><a class="reference internal" href="../chapter_2_ranking/5.multi_scenario/1.multi_tower.html">3.5.1. 多塔结构</a></li>
<li class="toctree-l3"><a class="reference internal" href="../chapter_2_ranking/5.multi_scenario/2.dynamic_weight.html">3.5.2. 动态权重建模</a></li>
<li class="toctree-l3"><a class="reference internal" href="../chapter_2_ranking/5.multi_scenario/3.summary.html">3.5.3. 小结</a></li>
</ul>
</li>
</ul>
</li>
<li class="toctree-l1"><a class="reference internal" href="../chapter_3_rerank/index.html">4. 重排模型</a><ul>
<li class="toctree-l2"><a class="reference internal" href="../chapter_3_rerank/1.greedy.html">4.1. 基于贪心的重排</a></li>
<li class="toctree-l2"><a class="reference internal" href="../chapter_3_rerank/2.personalized.html">4.2. 基于个性化的重排</a></li>
<li class="toctree-l2"><a class="reference internal" href="../chapter_3_rerank/3.summary.html">4.3. 本章小结</a></li>
</ul>
</li>
<li class="toctree-l1"><a class="reference internal" href="../chapter_4_trends/index.html">5. 难点及热点研究</a><ul>
<li class="toctree-l2"><a class="reference internal" href="../chapter_4_trends/1.debias.html">5.1. 模型去偏</a></li>
<li class="toctree-l2"><a class="reference internal" href="../chapter_4_trends/2.cold_start.html">5.2. 冷启动问题</a></li>
<li class="toctree-l2"><a class="reference internal" href="../chapter_4_trends/3.generative.html">5.3. 生成式推荐</a></li>
<li class="toctree-l2"><a class="reference internal" href="../chapter_4_trends/4.summary.html">5.4. 本章小结</a></li>
</ul>
</li>
<li class="toctree-l1"><a class="reference internal" href="../chapter_5_projects/index.html">6. 项目实践</a><ul>
<li class="toctree-l2"><a class="reference internal" href="../chapter_5_projects/1.understanding.html">6.1. 赛题理解</a></li>
<li class="toctree-l2"><a class="reference internal" href="../chapter_5_projects/2.baseline.html">6.2. Baseline</a></li>
<li class="toctree-l2"><a class="reference internal" href="../chapter_5_projects/3.analysis.html">6.3. 数据分析</a></li>
<li class="toctree-l2"><a class="reference internal" href="../chapter_5_projects/4.recall.html">6.4. 多路召回</a></li>
<li class="toctree-l2"><a class="reference internal" href="../chapter_5_projects/5.feature_engineering.html">6.5. 特征工程</a></li>
<li class="toctree-l2"><a class="reference internal" href="../chapter_5_projects/6.ranking.html">6.6. 排序模型</a></li>
</ul>
</li>
<li class="toctree-l1 current"><a class="reference internal" href="index.html">7. Appendix</a><ul class="current">
<li class="toctree-l2 current"><a class="current reference internal" href="#">7.1. Word2vec</a></li>
</ul>
</li>
</ul>
<ul>
<li class="toctree-l1"><a class="reference internal" href="../chapter_references/references.html">参考文献</a></li>
</ul>

            </nav>
        
        </div>
    
</header>
        <main class="mdl-layout__content" tabIndex="0">

	<script type="text/javascript" src="../_static/sphinx_materialdesign_theme.js "></script>
    <header class="mdl-layout__drawer">
    
          <!-- Title -->
      <span class="mdl-layout-title">
          <a class="title" href="../index.html">
              <span class="title-text">
                  FunRec 推荐系统
              </span>
          </a>
      </span>
    
    
      <div class="globaltoc">
        <span class="mdl-layout-title toc">Table Of Contents</span>
        
        
            
            <nav class="mdl-navigation">
                <ul>
<li class="toctree-l1"><a class="reference internal" href="../chapter_preface/index.html">前言</a></li>
<li class="toctree-l1"><a class="reference internal" href="../chapter_installation/index.html">安装</a></li>
<li class="toctree-l1"><a class="reference internal" href="../chapter_notation/index.html">符号</a></li>
</ul>
<ul class="current">
<li class="toctree-l1"><a class="reference internal" href="../chapter_0_introduction/index.html">1. 推荐系统概述</a><ul>
<li class="toctree-l2"><a class="reference internal" href="../chapter_0_introduction/1.intro.html">1.1. 推荐系统是什么？</a></li>
<li class="toctree-l2"><a class="reference internal" href="../chapter_0_introduction/2.outline.html">1.2. 本书概览</a></li>
</ul>
</li>
<li class="toctree-l1"><a class="reference internal" href="../chapter_1_retrieval/index.html">2. 召回模型</a><ul>
<li class="toctree-l2"><a class="reference internal" href="../chapter_1_retrieval/1.cf/index.html">2.1. 协同过滤</a><ul>
<li class="toctree-l3"><a class="reference internal" href="../chapter_1_retrieval/1.cf/1.itemcf.html">2.1.1. 基于物品的协同过滤</a></li>
<li class="toctree-l3"><a class="reference internal" href="../chapter_1_retrieval/1.cf/2.usercf.html">2.1.2. 基于用户的协同过滤</a></li>
<li class="toctree-l3"><a class="reference internal" href="../chapter_1_retrieval/1.cf/3.mf.html">2.1.3. 矩阵分解</a></li>
<li class="toctree-l3"><a class="reference internal" href="../chapter_1_retrieval/1.cf/4.summary.html">2.1.4. 总结</a></li>
</ul>
</li>
<li class="toctree-l2"><a class="reference internal" href="../chapter_1_retrieval/2.embedding/index.html">2.2. 向量召回</a><ul>
<li class="toctree-l3"><a class="reference internal" href="../chapter_1_retrieval/2.embedding/1.i2i.html">2.2.1. I2I召回</a></li>
<li class="toctree-l3"><a class="reference internal" href="../chapter_1_retrieval/2.embedding/2.u2i.html">2.2.2. U2I召回</a></li>
<li class="toctree-l3"><a class="reference internal" href="../chapter_1_retrieval/2.embedding/3.summary.html">2.2.3. 总结</a></li>
</ul>
</li>
<li class="toctree-l2"><a class="reference internal" href="../chapter_1_retrieval/3.sequence/index.html">2.3. 序列召回</a><ul>
<li class="toctree-l3"><a class="reference internal" href="../chapter_1_retrieval/3.sequence/1.user_interests.html">2.3.1. 深化用户兴趣表示</a></li>
<li class="toctree-l3"><a class="reference internal" href="../chapter_1_retrieval/3.sequence/2.generateive_recall.html">2.3.2. 生成式召回方法</a></li>
<li class="toctree-l3"><a class="reference internal" href="../chapter_1_retrieval/3.sequence/3.summary.html">2.3.3. 总结</a></li>
</ul>
</li>
</ul>
</li>
<li class="toctree-l1"><a class="reference internal" href="../chapter_2_ranking/index.html">3. 精排模型</a><ul>
<li class="toctree-l2"><a class="reference internal" href="../chapter_2_ranking/1.wide_and_deep.html">3.1. 记忆与泛化</a></li>
<li class="toctree-l2"><a class="reference internal" href="../chapter_2_ranking/2.feature_crossing/index.html">3.2. 特征交叉</a><ul>
<li class="toctree-l3"><a class="reference internal" href="../chapter_2_ranking/2.feature_crossing/1.second_order.html">3.2.1. 二阶特征交叉</a></li>
<li class="toctree-l3"><a class="reference internal" href="../chapter_2_ranking/2.feature_crossing/2.higher_order.html">3.2.2. 高阶特征交叉</a></li>
<li class="toctree-l3"><a class="reference internal" href="../chapter_2_ranking/2.feature_crossing/3.summary.html">3.2.3. 总结</a></li>
</ul>
</li>
<li class="toctree-l2"><a class="reference internal" href="../chapter_2_ranking/3.sequence.html">3.3. 序列建模</a></li>
<li class="toctree-l2"><a class="reference internal" href="../chapter_2_ranking/4.multi_objective/index.html">3.4. 多目标建模</a><ul>
<li class="toctree-l3"><a class="reference internal" href="../chapter_2_ranking/4.multi_objective/1.arch.html">3.4.1. 基础结构演进</a></li>
<li class="toctree-l3"><a class="reference internal" href="../chapter_2_ranking/4.multi_objective/2.dependency_modeling.html">3.4.2. 任务依赖建模</a></li>
<li class="toctree-l3"><a class="reference internal" href="../chapter_2_ranking/4.multi_objective/3.multi_loss_optim.html">3.4.3. 多目标损失融合</a></li>
<li class="toctree-l3"><a class="reference internal" href="../chapter_2_ranking/4.multi_objective/4.summary.html">3.4.4. 小结</a></li>
</ul>
</li>
<li class="toctree-l2"><a class="reference internal" href="../chapter_2_ranking/5.multi_scenario/index.html">3.5. 多场景建模</a><ul>
<li class="toctree-l3"><a class="reference internal" href="../chapter_2_ranking/5.multi_scenario/1.multi_tower.html">3.5.1. 多塔结构</a></li>
<li class="toctree-l3"><a class="reference internal" href="../chapter_2_ranking/5.multi_scenario/2.dynamic_weight.html">3.5.2. 动态权重建模</a></li>
<li class="toctree-l3"><a class="reference internal" href="../chapter_2_ranking/5.multi_scenario/3.summary.html">3.5.3. 小结</a></li>
</ul>
</li>
</ul>
</li>
<li class="toctree-l1"><a class="reference internal" href="../chapter_3_rerank/index.html">4. 重排模型</a><ul>
<li class="toctree-l2"><a class="reference internal" href="../chapter_3_rerank/1.greedy.html">4.1. 基于贪心的重排</a></li>
<li class="toctree-l2"><a class="reference internal" href="../chapter_3_rerank/2.personalized.html">4.2. 基于个性化的重排</a></li>
<li class="toctree-l2"><a class="reference internal" href="../chapter_3_rerank/3.summary.html">4.3. 本章小结</a></li>
</ul>
</li>
<li class="toctree-l1"><a class="reference internal" href="../chapter_4_trends/index.html">5. 难点及热点研究</a><ul>
<li class="toctree-l2"><a class="reference internal" href="../chapter_4_trends/1.debias.html">5.1. 模型去偏</a></li>
<li class="toctree-l2"><a class="reference internal" href="../chapter_4_trends/2.cold_start.html">5.2. 冷启动问题</a></li>
<li class="toctree-l2"><a class="reference internal" href="../chapter_4_trends/3.generative.html">5.3. 生成式推荐</a></li>
<li class="toctree-l2"><a class="reference internal" href="../chapter_4_trends/4.summary.html">5.4. 本章小结</a></li>
</ul>
</li>
<li class="toctree-l1"><a class="reference internal" href="../chapter_5_projects/index.html">6. 项目实践</a><ul>
<li class="toctree-l2"><a class="reference internal" href="../chapter_5_projects/1.understanding.html">6.1. 赛题理解</a></li>
<li class="toctree-l2"><a class="reference internal" href="../chapter_5_projects/2.baseline.html">6.2. Baseline</a></li>
<li class="toctree-l2"><a class="reference internal" href="../chapter_5_projects/3.analysis.html">6.3. 数据分析</a></li>
<li class="toctree-l2"><a class="reference internal" href="../chapter_5_projects/4.recall.html">6.4. 多路召回</a></li>
<li class="toctree-l2"><a class="reference internal" href="../chapter_5_projects/5.feature_engineering.html">6.5. 特征工程</a></li>
<li class="toctree-l2"><a class="reference internal" href="../chapter_5_projects/6.ranking.html">6.6. 排序模型</a></li>
</ul>
</li>
<li class="toctree-l1 current"><a class="reference internal" href="index.html">7. Appendix</a><ul class="current">
<li class="toctree-l2 current"><a class="current reference internal" href="#">7.1. Word2vec</a></li>
</ul>
</li>
</ul>
<ul>
<li class="toctree-l1"><a class="reference internal" href="../chapter_references/references.html">参考文献</a></li>
</ul>

            </nav>
        
        </div>
    
</header>

    <div class="document">
        <div class="page-content" role="main">
        
  <section id="word2vec">
<span id="id1"></span><h1><span class="section-number">7.1. </span>Word2vec<a class="headerlink" href="#word2vec" title="Permalink to this heading">¶</a></h1>
<p>在自然语言处理（NLP）任务中，如何将单词转换为能够被机器直接处理的数值形式一直是一个根本问题。早期的方法（例如
one-hot
编码）虽然直观，但存在向量维度高、稀疏且无法体现单词间细微语义差异的问题。我们迫切需要一种既低维又能捕捉单词语义和语法信息的表示方式。Word2vec
<span id="id2">(<a class="reference internal" href="../chapter_references/references.html#id15" title="Mikolov, T., Sutskever, I., Chen, K., Corrado, G. S., &amp; Dean, J. (2013). Distributed representations of words and phrases and their compositionality. Advances in neural information processing systems, 26.">Mikolov <em>et al.</em>, 2013</a>)</span>
的目标在于通过大量无标签文本数据学习每个单词的<strong>dense vector
表示</strong>，使得：</p>
<ul class="simple">
<li><p>在低维空间中，语义相近的单词其向量距离也较为接近；</p></li>
<li><p>单词之间的关系可以通过向量运算反映出其内在联系。</p></li>
</ul>
<p>这样得到的词向量可以直接应用于文本分类、机器翻译、信息检索等下游任务中。</p>
<p><strong>Word2vec概述</strong></p>
<p>Word2vec
通过上下文（context）来学习单词的向量表示。文本的上下文为现代语言学中的一个基本概念，由
<span id="id3">(<a class="reference internal" href="../chapter_references/references.html#id16" title="Firth, J. R. (1957). Studies in Linguistic Analysis. Blackwell.">Firth, 1957</a>)</span> 提出：“You shall know a word by the
company it keeps.”
这一假设指导我们通过观察单词在大量文本中与哪些词共同出现，来推断该单词的语义。具体来说，当单词出现在文本中时，其上下文是该单词周围的一些单词。例如下图：</p>
<figure class="align-default" id="id5">
<img alt="../_images/w2v_example.svg" src="../_images/w2v_example.svg" /><figcaption>
<p><span class="caption-number">图7.1.1 </span><span class="caption-text">上下文示例</span><a class="headerlink" href="#id5" title="Permalink to this image">¶</a></p>
</figcaption>
</figure>
<p>这些上下文单词决定了单词<code class="docutils literal notranslate"><span class="pre">loves</span></code>的语义。Word2vec
使用了大量的文本语料库来学习单词的向量表示。它遍历语料库中的每个单词，并通过调整单词的向量表示，使得模型预测的上下文单词与语料库中的真实上下文单词尽可能一致。</p>
<p>Word2vec 主要包含两种模型：<strong>Skip-gram</strong> 和 <strong>Continuous Bag of Words
model (CBOW)</strong>。</p>
<p><strong>Skip-gram 模型</strong> <img alt="skip-gram" src="../_images/w2v_skip_gram.svg" /></p>
<p>Skip-gram
模型通过中心词预测上下文词。具体来说，给定一个中心词，模型预测其上下文词
出现的概率。如果我们考虑在语料库中的滑行窗口，在位置 <span class="math notranslate nohighlight">\(t\)</span>
的中心词为 <span class="math notranslate nohighlight">\(w_t\)</span>，其上下文词为
<span class="math notranslate nohighlight">\(w_{t-m}, w_{t-m+1}, \cdots, w_{t-1}, w_{t+1}, \cdots, w_{t+m}\)</span>，其中
<span class="math notranslate nohighlight">\(m\)</span> 是窗口大小。Skip-gram 中上下文词的条件概率
<span class="math notranslate nohighlight">\(P(w_{t+j} | w_t)\)</span> 可以表示为：</p>
<div class="math notranslate nohighlight" id="equation-skip-gram-likelihood">
<span class="eqno">(7.1.1)<a class="headerlink" href="#equation-skip-gram-likelihood" title="Permalink to this equation">¶</a></span>\[P(w_{t+j} | w_t) = \frac{e^{v_{w_{t+j}}^T v_{w_t}}}{\sum_{k=1}^{V} e^{v_{w_k}^T v_{w_t}}}\]</div>
<p>其中 <span class="math notranslate nohighlight">\(v_{w_i}\)</span> 是单词 <span class="math notranslate nohighlight">\(w_i\)</span>
的向量表示。那么，将滑行窗口遍历整个语料库，我们可以得到似然函数：</p>
<div class="math notranslate nohighlight" id="equation-chapter-appendix-word2vec-0">
<span class="eqno">(7.1.2)<a class="headerlink" href="#equation-chapter-appendix-word2vec-0" title="Permalink to this equation">¶</a></span>\[\prod_{t=1}^{T} \prod_{m&lt;=j&lt;=m} P(w_{t+j} | w_t)\]</div>
<p><strong>CBOW 模型</strong> <img alt="cbow" src="../_images/w2v_cbow.svg" /> CBOW
模型通过上下文词预测中心词。具体来说，给定上下文词
<span class="math notranslate nohighlight">\(o\)</span>，模型预测中心词 <span class="math notranslate nohighlight">\(c\)</span> 出现的概率。CBOW
模型中心词的条件概率 <span class="math notranslate nohighlight">\(P(w_t | w_{t+j})\)</span> 可以表示为：</p>
<div class="math notranslate nohighlight" id="equation-cbow-likelihood">
<span class="eqno">(7.1.3)<a class="headerlink" href="#equation-cbow-likelihood" title="Permalink to this equation">¶</a></span>\[P(w_t | w_{t+j}) = \frac{e^{v_{w_t}^T v_{w_{t+j}}}}{\sum_{k=1}^{V} e^{v_{w_k}^T v_{w_{t+j}}}}\]</div>
<p>将滑行窗口遍历整个语料库，我们可以得到似然函数：</p>
<div class="math notranslate nohighlight" id="equation-chapter-appendix-word2vec-1">
<span class="eqno">(7.1.4)<a class="headerlink" href="#equation-chapter-appendix-word2vec-1" title="Permalink to this equation">¶</a></span>\[\prod_{t=1}^{T} \prod_{m&lt;=j&lt;=m} P(w_t | w_{t+j})\]</div>
<p><strong>Word2vec 的模型结构</strong></p>
<p>在以上条件概率公式中，容易引起误会的是 中心词的向量 <span class="math notranslate nohighlight">\(v_{w_t}\)</span> 和
上下文词的向量 <span class="math notranslate nohighlight">\(v_{w_{t+j}}\)</span> 实际并不在一个向量空间中
<span id="id4">(<a class="reference internal" href="../chapter_references/references.html#id17" title="Rong, X. (2014). Word2vec parameter learning explained. arXiv preprint arXiv:1411.2738.">Rong, 2014</a>)</span>。以 Skip-gram
模型为例，假设向量空间的维度为<span class="math notranslate nohighlight">\(D\)</span>，中心词向量表<span class="math notranslate nohighlight">\(\mathbf{W} \in \mathbb{R}^{V \times D}\)</span>，上下文词向量表<span class="math notranslate nohighlight">\(\mathbf{W}^c \in \mathbb{R}^{V \times D}\)</span>，其中<span class="math notranslate nohighlight">\(V\)</span>是词汇表的大小。Word2vec
的模型结构如下：</p>
<figure class="align-default" id="id6">
<a class="reference internal image-reference" href="../_images/w2v_model.svg"><img alt="../_images/w2v_model.svg" src="../_images/w2v_model.svg" width="500px" /></a>
<figcaption>
<p><span class="caption-number">图7.1.2 </span><span class="caption-text">word2vec 模型</span><a class="headerlink" href="#id6" title="Permalink to this image">¶</a></p>
</figcaption>
</figure>
<p>从左到右，输入给定一个中心词的one-hot表示
<span class="math notranslate nohighlight">\(\mathbf{x}_t \in \{0, 1\}^V\)</span>，中心词的向量
<span class="math notranslate nohighlight">\(\mathbf{w}_{t}\)</span> 可以表示为：</p>
<div class="math notranslate nohighlight" id="equation-chapter-appendix-word2vec-2">
<span class="eqno">(7.1.5)<a class="headerlink" href="#equation-chapter-appendix-word2vec-2" title="Permalink to this equation">¶</a></span>\[\mathbf{v}_{w_t} = \mathbf{x}_t^T \mathbf{W}\]</div>
<p>再将中心词的向量 <span class="math notranslate nohighlight">\(\mathbf{v}_{w_t}\)</span> 与 上下文向量词表
<span class="math notranslate nohighlight">\(\mathbf{W}^o\)</span>
第<span class="math notranslate nohighlight">\(t+j\)</span>行相乘，得到softmax的分母的输入。而分子输入为上下文向量词表
<span class="math notranslate nohighlight">\(\mathbf{W}^o\)</span> 整个与中心词向量 <span class="math notranslate nohighlight">\(\mathbf{v}_{w_t}\)</span>
的乘积。最后，通过softmax, 可以对上下文词的概率进行预测。</p>
<p><strong>负采样</strong></p>
<p>在计算softmax <a class="reference internal" href="#equation-skip-gram-likelihood">(7.1.1)</a> 和
<a class="reference internal" href="#equation-cbow-likelihood">(7.1.3)</a>
时，需要对词汇表中的所有单词进行计算，这会导致计算复杂度非常高。为了降低计算复杂度，Word2vec
采用了负采样技术。以 Skip-gram
模型为例，负采样技术通过随机采样一些负样本（即，不是中心词的上下文词），来替代
<span class="math notranslate nohighlight">\(\log P(w_{t+j} | w_t)\)</span>。</p>
<div class="math notranslate nohighlight" id="equation-chapter-appendix-word2vec-3">
<span class="eqno">(7.1.6)<a class="headerlink" href="#equation-chapter-appendix-word2vec-3" title="Permalink to this equation">¶</a></span>\[\log \sigma(v_{w_{t+j}}^T v_{w_t}) + \sum_{i=1}^{k} \mathbb{E}_{w_i \sim P_n(w)} \log \sigma(-v_{w_i}^T v_{w_t})\]</div>
<p>其中，<span class="math notranslate nohighlight">\(\sigma(x) = \frac{1}{1 + e^{-x}}\)</span> 是 sigmoid
函数，<span class="math notranslate nohighlight">\(k\)</span> 是负样本的数量，<span class="math notranslate nohighlight">\(P_n(w)\)</span>
是负样本的分布。在原文中，<span class="math notranslate nohighlight">\(P_n(w) = \frac{\text{count}(w)^{3/4}}{\sum_{\text{all words } w'} \text{count}(w')^{3/4}}\)</span>,
<span class="math notranslate nohighlight">\(\text{count}(w)\)</span> 是单词 <span class="math notranslate nohighlight">\(w\)</span> 在语料库中出现的次数。</p>
<p>由函数单调性可知，<span class="math notranslate nohighlight">\(\log \sigma(v_{w_{t+j}}^T v_{w_t})\)</span>
越大，<span class="math notranslate nohighlight">\(\log \sigma(-v_{w_i}^T v_{w_t})\)</span> 越小，即
<span class="math notranslate nohighlight">\(P(w_{t+j} | w_t)\)</span>
越大。直观上，极大似然会将经常出现的上下文词拉向中心词，而负采样则将负样本推离中心词。这与原始的似然函数
<a class="reference internal" href="#equation-skip-gram-likelihood">(7.1.1)</a>
的优化目标一致，并且在最大化原始似然函数的同时
<span class="math notranslate nohighlight">\(\max P(w_{t+j} | w_t)\)</span>, 并且避免了对整个词汇表的计算。</p>
</section>


        </div>
        <div class="side-doc-outline">
            <div class="side-doc-outline--content"> 
            </div>
        </div>

      <div class="clearer"></div>
    </div><div class="pagenation">
     <a id="button-prev" href="index.html" class="mdl-button mdl-js-button mdl-js-ripple-effect mdl-button--colored" role="botton" accesskey="P">
         <i class="pagenation-arrow-L fas fa-arrow-left fa-lg"></i>
         <div class="pagenation-text">
            <span class="pagenation-direction">Previous</span>
            <div>7. Appendix</div>
         </div>
     </a>
     <a id="button-next" href="../chapter_references/references.html" class="mdl-button mdl-js-button mdl-js-ripple-effect mdl-button--colored" role="botton" accesskey="N">
         <i class="pagenation-arrow-R fas fa-arrow-right fa-lg"></i>
        <div class="pagenation-text">
            <span class="pagenation-direction">Next</span>
            <div>参考文献</div>
        </div>
     </a>
  </div>
        
        </main>
    </div>
  </body>
</html>